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Abstract 


CAE  Professional  Services  (Canada)  Inc.  (CAE  PS)  was  contracted  by  Defence  Research  and 
Development  Canada  (DRDC)  Toronto  to  conduct  work  in  accordance  with  a  Call-up  under  Contract 
Number  W771 1-0681 1 1.  The  intent  of  the  Call-up  was  to  provide  experimental  support  to  DRDC 
Toronto  in  the  evaluation  of  the  Victoria  Class  Virtual  Submarine  (VCVS);  DRDC  Toronto  designed  the 
experiment  and  CAE  PS  assisted  in  the  execution  of  the  project.  The  evaluation  of  the  VCVS  was 
conducted  in  accordance  with  Protocol  Number  L-809  (Magee,  Cain,  &  Thompson,  2011),  an 
experimental  protocol  approved  by  the  DRDC  Human  Research  Ethics  Committee  (HREC).The  VCVS  is 
a  game-based  software  program  designed  to  deliver  training  to  military  personnel  in  a  simulated 
submarine  environment.  This  document  describes  the  implementation  of  the  protocol,  provides  qualitative 
findings  and  identifies  the  limitations  and  lessons  learned  from  the  conduct  of  the  work.  The  results  of  the 
behavioural  studies  are  reported  separately  by  DRDC  Toronto. 


Resume 


Recherche  et  developpement  pour  la  defense  Canada  (RDDC)  Toronto  a  attribue  un  contrat  a  CAE 
Services  professionnels  (Canada)  Inc.  (CAE  SP)  en  vue  d’effectuer  le  travail  conformement  a  la 
commande  du  numero  de  contrat  W771 1-0681 1 1.  L’objectif  de  cette  commande  etait  de  foumir  un 
soutien  experimental  a  RDDC  Toronto  dans  le  cadre  de  revaluation  du  sous-marin  virtuel  de  la  classe 
VICTORIA  (SVCV);  RDDC  Toronto  a  congu  P  experience  et  CAE  SP  a  participe  a  la  realisation  du 
projet.  L’evaluation  du  SVCV  a  ete  menee  conformement  au  numero  de  protocole  L-809  (Magee,  Cain  et 
Thompson,  2011)  :  un  protocole  experimental  approuve  par  le  Comite  d’ethique  en  matiere  d’etude  sur 
des  sujets  humains  (CEESH)  de  RDDC.  Le  SVCV  est  un  logiciel  base  sur  le  jeu  conpu  afin  de  donner  de 
la  formation  aux  militaires  dans  un  environnement  de  sous-marin  virtuel.  Le  present  document  decrit  la 
mise  en  oeuvre  du  protocole,  presente  les  resultats  qualitatifs  et  determine  les  limites  et  les  logons  retenues 
de  la  realisation  du  travail.  Les  resultats  des  etudes  sur  le  comportement  font  l’objet  d’un  rapport  distinct 
de  RDDC  Toronto. 
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Executive  summary 


Experimental  support  for  evaluation  of  the  Victoria  Class  Virtual 
Submarine  (VCVS) 

Lochlan  E.  Magee;  DRDC  Toronto  CR  2012-013;  Defence  R&D  Canada  -  Toronto; 

March  2012. 

Introduction  or  background:  The  Canadian  Navy  eLearning  Centre  of  Expertise  (NeLCoE) 
developed  the  Canadian  Virtual  Naval  Fleet  (CVNF),  a  game-based,  desk-top,  virtual 
environment  for  training  maritime  procedures  and  for  familiarizing  trainees  with  the  spatial 
structures  of  large  naval  vessels  that  are  often  unavailable  for  on-board  training.  The  Director 
Maritime  Training  and  Education  (DMTE)  requested  an  objective  investigation  of  the  training 
effectiveness  of  Victoria  Class  Virtual  Submarine  (VCVS),  one  of  the  first  implementations  of 
the  CVNF.  Defence  R&D  Canada  (DRDC)  -  Toronto  engaged  CAE  Professional  Services 
(Canada)  Inc.  to  implement  an  experimental  protocol  that  was  conceived  by  DRDC  Toronto.  The 
main  purposes  of  the  experimental  protocol  were  to  gain  behavioural  information  about  the 
training  effectiveness  of  the  VCVS  and  to  determine  the  need  for  further  development  of  the 
hardware  or  software.  A  complex  task  was  used  for  the  assessment.  It  involved  the  isolation  of  a 
bulkhead,  which  is  performed  if  a  fire  or  flood  occurs  within  the  submarine.  Qualified 
submariners  must  know  the  locations,  names,  functions,  and  operations  of  the  valves  and  tools 
that  are  needed  to  perfonn  this  drill.  Participants  learned  and  performed  the  task  to  criterion 
within  the  VCVS,  aboard  HMCS  Comer  Brook,  or  both. 

Results:  This  report  (1)  describes  how  the  experimental  protocol  was  implemented,  (2)  provides 
summary  data  that  describe  the  participants,  (3)  includes  observations  made  by  the  experimenters 
during  the  conduct  of  the  study,  (4)  provides  anecdotal  reports  made  by  the  participants,  (5) 
identifies  deficiencies  in  the  simulation  and  their  impact  on  performance,  (6)  provides  a  method 
and  specific  recommendations  for  correcting  deficiencies  with  the  simulation,  (7)  provides 
observations  that  could  affect  the  interpretation  of  the  results  of  the  behavioural  analyses,  which 
are  reported  separately,  and  (8)  offers  first-hand  observations  and  opinions  about  the  usefulness 
of  the  VCVS. 

Significance:  The  report  provides  assurance  that  the  experimental  protocol  was  implemented  as 
planned.  It  also  provides  observations  helpful  to  the  interpretation  of  the  behavioural  results 
(reported  separately),  guidance  for  improving  the  VCVS  for  future  use  by  the  Royal  Canadian 
Navy,  and  lessons-learned  for  future  behavioural  studies  of  this  type. 
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Experimental  support  for  evaluation  of  the  Victoria  Class  Virtual 
Submarine  (VCVS) 

Lochlan  E.  Magee  ;  DRDC  Toronto  CR  2012-013  ;  R  &  D  pour  la  defense  Canada  - 

Toronto;  mars  2012. 

Introduction  ou  contexte  :  L’equipe  du  Centre  d’expertise  de  l’apprentissage  en  ligne  de  la 
Marine  canadienne  a  developpe  une  flotte  navale  virtuelle  canadienne  (FNVC),  c’est-a-dire  un 
environnement  virtuel  pour  ordinateur  base  sur  le  jeu,  a  des  fins  de  procedures  de  formation 
maritime  et  pour  familiariser  les  stagiaires  avec  les  structures  spatiales  de  larges  navires  qui  ne 
sont  pas  toujours  disponibles  pour  pouvoir  y  donner  de  la  formation  a  bord.  Le  Directeur  - 
Instruction  et  education  maritimes  (DIEM)  a  demande  la  tenue  d’une  enquete  objective  de 
Fefficacite  de  la  formation  avec  le  sous-marin  virtuel  de  la  classe  VICTORIA  (SVCV),  Tune  des 
premieres  initiatives  de  la  FNVC.  R  &  D  pour  la  defense  Canada  (RDDC)  Toronto  a  demande  a 
CAE  Services  professionnels  (Canada)  Inc.  de  mettre  en  place  un  protocole  experimental  congu 
par  RDDC  Toronto.  Les  objectifs  principaux  du  protocole  experimental  etaient  d’acquerir  de 
l’information  sur  le  comportement  concernant  Fefficacite  de  la  formation  avec  le  SVCV,  ainsi 
que  de  determiner  s’il  est  necessaire  de  pousser  le  developpement  du  materiel  ou  du  logiciel.  Une 
tache  complexe  a  ete  realisee  pour  F  evaluation.  Pour  ce  faire,  il  a  fallu  isoler  une  cloison;  il  est 
possible  de  le  faire  si  un  incendie  ou  une  inondation  se  produit  dans  le  sous-marin.  Les  sous- 
mariniers  qualifies  doivent  connaitre  les  emplacements,  les  noms,  les  fonctions  et  les  operations 
des  valves  et  des  outils  qui  sont  necessaires  aux  fins  de  l’exercice.  Les  participants  ont  tire  des 
lemons  et  ont  effectue  la  tache  en  fonction  des  criteres  du  SVCV,  a  bord  du  NCSM  CORNER 
BROOK,  ou  les  deux. 

Resultats  :  Le  present  rapport  (1)  decrit  comment  le  protocole  experimental  a  ete  mis  en  oeuvre, 
(2)  fournit  des  donnees  sommaires  decrivant  les  participants,  (3)  comprend  les  observations  des 
experimentateurs  pendant  la  tenue  de  F  etude,  (4)  presente  les  rapports  isoles  des  participants,  (5) 
releve  les  lacunes  liees  a  la  simulation  et  leurs  repercussions  sur  le  rendement,  (6)  fournit  une 
methode  et  des  recommandations  precises  pour  corriger  les  lacunes  liees  a  la  simulation,  (7) 
indique  des  observations  qui  pourrait  avoir  des  repercussions  sur  F interpretation  des  resultats  des 
analyses  comportementales  (sujet  d’un  rapport  distinct),  et  (8)  presente  des  observations  et  des 
opinions  de  premiere  main  sur  l’utilite  du  SVCV. 

Importance  :  Ce  rapport  fournit  Fassurance  que  le  protocole  experimental  a  ete  mis  en  oeuvre 
comme  prevu.  De  plus,  il  presente  des  observations  utiles  en  vue  de  F interpretation  des  resultats 
comportementaux  (sujet  d’un  rapport  distinct),  oriente  F  amelioration  du  SVCV  a  des  fins 
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d’utilisation  future  par  la  Marine  royale  canadienne,  puis  indique  les  logons  retenues  en  vue 
d’autres  etudes  comportementales  de  ce  genre. 
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1.  Introduction 

The  Royal  Canadian  Navy  (RCN)  faces  increasing  challenges  and  setbacks  to  deliver  Victoria  Class 
Submarine  training  to  naval  personnel.  These  training  challenges  are  augmented  by  limitations  to  gaining 
physical  access  to  the  Victoria  Class  Submarines.  To  address  these  challenges  the  Canadian  Navy 
eLearning  Centre  of  Expertise  (NeLCoE)  developed  the  Victoria  Class  Virtual  Submarine  (VCVS);  the 
VCVS  is  a  game-based  software  program  designed  to  deliver  training  to  naval  personnel  in  a  simulated 
environment.  The  objective  of  the  VCVS  is  to  train  personnel  on  the  spatial  layout  of  the  submarine  and 
the  conduct  of  onboard  procedural  tasks. 

The  Director  Maritime  Training  and  Education  (DMTE)  requested  support  from  Defence  Research  and 
Development  Canada  (DRDC)  Toronto  to  design  and  conduct  a  scientific  study  to  evaluate  the  VCVS  as 
a  future  training  platform  for  RCN  personnel.  DRDC  Toronto  contracted  CAE  Professional  Services 
(Canada)  Inc.  (CAE  PS)  to  provide  additional  experimental  support  during  the  execution  of  the  study. 
Over  a  four  week  period,  CAE  PS  provided  assistance  with  conducting  the  experiment  aboard  Her 
Majesty’s  Canadian  Ship  (HMCS)  Comer  Brook,  while  alongside  at  Her  Majesty’s  Canadian  (HMC) 
Dockyard  Esquimalt,  and  in  a  classroom  at  the  Canadian  Forces  Base  (CEB)  Esquimalt,  British 
Columbia.  The  experiment  was  conducted  in  accordance  with  Protocol  Number  L-809  (Magee,  Cain,  & 
Thompson,  2011)  which  received  formal  approval  from  the  DRDC  Human  Research  Ethics  Committee 
(HREC). 

Scope 

This  technical  report  is  the  deliverable  prepared  in  accordance  with  the  requirements  for  Call-up.  This 
document  describes  the  implementation  of  the  protocol,  quantitative  and  qualitative  findings  and 
identifies  the  limitations  and  lessons  learned  from  the  conduct  of  the  work.  The  behavioural  results  are 
reported  more  fully  by  DRDC  Toronto  in  a  technical  report. 
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2.  pre-Experiment  Tasks 

In  providing  experimental  support  to  DRDC  Toronto  during  the  evaluation  of  the  VCVS,  CAE  PS 

executed  a  series  of  general  tasks  throughout  the  conduct  of  this  project  (see  Annex  A).  In  addition,  a 

series  of  pre-experimental  tasks  was  completed  prior  to  executing  the  testing  sessions  aboard  HMCS 

Comer  Brook. 

Pre-experimental  tasks  executed  aboard  HMCS  Comer  Brook  included: 

a)  Signing  in  with  HMCS  Corner  Brook  Commissionaire,  prior  to  boarding  HMCS 
Corner  Brook; 

b)  Notifying  HMCS  Corner  Brook  Duty  Watch  Supervisor  (DWS)  of  physical  presence 
aboard  HMCS  Corner  Brook; 

c)  Signing  the  experimenter  and  all  participants  in  and  out  on  HMCS  Comer  Brook  accountability  sheet; 
and, 

d)  Requesting  a  qualified  submariner  perform  safety  briefings  on  the  use  of  the  Emergency  Breathing 
System  (EBS)  and  the  Emergency  Escape  Breathing  Device  (EEBD)  prior  to  his  participation  in  the 
study  aboard  HCMS  Comer  Brook. 
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3.  methodology 

The  following  sub-sections  describe  the  implementation  of  the  protocol. 

Study  Participants 

The  following  section  describes  the  characteristics  of  each  of  the  participant  groups  included  in  the  VCVS 
study.  The  RCN  personnel  who  participated  in  the  study  were  divided  into  three  testing  groups  based 
upon  their  prior  experience  task-qualified  credentials. 

Qualified,  Experimental  and  Control  Participants 

Participants  with  very  little  or  no  experience  working  aboard  submarines  were  assigned  to  either  the 
Control  or  Experimental  groups;  none  of  these  participants  had  experience  with  isolating  bulkhead  35. 

The  Control  group  (N=10  males;  mean  age  =  28.5  years)  and  the  Experimental  group  (N=10  males;  mean 
age  23.4  years)  consisted  of  either  Personnel  Awaiting  Training  (PATs)  or  RCN  non-commissioned 
members.  The  third  testing  group,  identified  as  the  Qualified  group,  consisted  of  experienced  and  task- 
qualified  submariners  (N=10  males;  mean  age  =  39.6  years).  Table  3-1  presents  the  age-related 
descriptive  data  for  each  of  the  participant  groups.  The  Personnel  Coordination  Centre  of  Maritime  Forces 
Pacific  recruited  available  participants  from  various  local  units. 

Table  3-1:  Descriptive  Statistics  upon  Participant  Age  Categorized  by  Group 


Age 

Years) 

Group 

Mean 

SD 

Maximum 

Minimum 

Qualified 

39.6 

8.5 

50 

26 

Experimental 

23.4 

6.5 

35 

18 

Control 

28.5 

9.6 

49 

20 

In  addition  to  providing  their  age,  participants  provided  service  experience  and  video  game  experience. 
With  respect  to  service  experience,  all  participants  were  asked  to  provide  the  total  number  of  years  they 
have  served  on  both  submarines  and/or  ships.  Table  3-2  and  Table  3-3  present  data  pertaining  to  service 
experience  (submarine  and  ship)  for  each  participant  group.  These  tables  demonstrate  that  Experimental 
and  Control  participants  in  this  study  had  little  to  no  experience  aboard  submarines  and  ships.  The 
experience  was  measured  in  years  for  the  Qualified  group,  however,  the  experience  was  measured  in 
hours  for  the  Control  and  Experimental  groups. 


Table  3-2:  Descriptive  Statistics  for  Qualified  Participant  Service  Experience 
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Qualified  9.7  7.4  25  1.5  3.4  3.0  10  0 


Table  3-3:  Descriptive  Statistics  for  Experimental  and  Control  Participant  Sendee  Experience 


Service  Experience 

Group 

Submarine  (Total  Number  of  Hours) 

Ship  (Total  Number  of  Weeks) 

Mean 

SD 

Max 

Min 

Mean 

SD 

Max 

Min 

Experimental 

0.03 

0.10 

0.33 

0.00 

12.02 

32.36 

104.00 

0.00 

Control 

1.71 

5.03 

16.00 

0.00 

37.21 

114.83 

364.00 

0.00 

Group  mean  video  game  experience  data  are  presented  below  in  Table  3-4.  These  results  demonstrate 
that  overall  the  Qualified  participants  were  the  most  experienced  video  game  players.  However,  on 
average  the  Experimental  participants  tended  to  have  more  video  game  experience  on  a  weekly  basis 
compared  to  the  other  two  groups.1 

Table  3-4:  Descriptive  Statistics  for  Participant  Video  Game  Experience  Categorized  by  Group 


Video  Game  Experience 

Group  Total  Number  of  Years  Number  of  Hours/ Week  Playing 


Mean 

SD 

Max 

Min 

Mean 

SD 

Max 

Min 

Qualified 

12.9 

10.9 

30 

0 

3.5 

7.0 

20 

0 

Experimental 

8.9 

7.8 

25 

0 

5.6 

6.3 

17.5 

0 

Control 

10.1 

8.1 

22 

0 

3.2 

4.3 

10 

0 

Materials  &  Design 

In  order  to  evaluate  the  effectiveness  of  the  VCVS  in  training  personnel  on  the  spatial  layout  and 
procedural  tasks  associated  with  Victoria  Class  submarines,  two  experiments  were  conducted  which 
required  participants  to  isolate  bulkhead  35  in  the  simulated  VCVS  environment  and/  or  aboard  HMCS 
Comer  Brook. 

The  first  experiment  was  designed  to  assess  reverse  transfer  of  training  effects  that  might  be  associated 
with  the  VCVS  by  comparing  the  number  of  trials  required  by  the  Qualified  and  Experimental 
participants  to  successfully  isolate  bulkhead  35  one  time  without  error  in  the  VCVS  (highlighted  as  blue 
in  Table  3-5  below). 

The  second  experiment,  designed  to  assess  forward  transfer  of  training  effects  associated  with  the  VCVS, 
compared  the  number  of  trials  Experimental  and  Control  participants  required  to  isolate  bulkhead  35  one 
time  perfectly  aboard  HMCS  Comer  Brook  (highlighted  as  green  in  Table  3-5  below).  Table  3-5  below 
describes  the  experimental  design  for  all  three  participant  groups.  The  Qualified  Group  participated  only 


1  Each  participant  provided  an  estimate  to  reflect  his  overall  experience  playing  video  games  and  his  experience 
playing  on  a  weekly  basis.  The  mean  level  of  experience  for  each  element  was  calculated  on  the  basis  of  these 
estimated  ranges. 
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in  the  VC  VS  familiarization  and  skill  acquisition  session  as  they  had  previously  achieved  their 
qualifications  to  perform  these  tasks  aboard  the  submarine. 

Table  3-5:  Experimental  Design  Associated  with  the  Study 


Experimental  Design 

Session  1 

Session  2 

Group 

VCVS  Familiarization 
and  Skill  Acquisition 
Session 

Submarine 

Familiarization  Session 

Submarine  Skill  Acquisition 
Session 

Control 

X 

X 

Experimental 

X 

X 

Qualified 

X 

During  all  skill  acquisition  trials,  compartment  (forward  or  aft)  and  environmental  (VC VS  or  submarine) 
specific  scoring  sheets  (See  Annex  B,  C,  D  and  E),  designed  by  the  experimenters,  were  used  to  record 
each  participant’s  spatial  and  procedural  task  performance  associated  with  isolating  bulkhead  35.  To 
successfully  isolate  bulkhead  35,  participants  had  to  correctly  locate,  identify,  explain  the  function  and 
operation  of  six  valves  associated  with  isolating  bulkhead  35  one  time  without  error  from  either  the 
forward  or  aft  compartment.  Participants  were  instructed  that  the  order  in  which  they  shut  the  six  valves 
was  unimportant,  but  that  they  must  take  the  most  direct  route  possible  for  a  successful  completion. 
Throughout  the  skill  acquisition  trials,  participants  were  allowed  to  ask  for  assistance  and  were  given 
immediate  corrective  feedback  from  the  experimenters  when  an  error  was  committed.  In  order  to  gain 
consensus  upon  the  scoring  of  spatial  and  procedural  tasks  associated  with  isolating  bulkhead  35,  the 
experimenters  (C.  Kersten  and  Dr.  A.  Thompson)  collectively  evaluated  the  first  four  Experimental 
participants  and  all  1 0  Qualified  participants. 

Procedure 

After  briefing  participants  on  their  rights,  roles  and  responsibilities  associated  with  their  participation  in 
the  VCVS  experiment,  participants  were  asked  to  complete  the  consent  form  and  were  given  the 
opportunity  to  read  the  Protocol  Number  L-809  (Magee,  Cain,  &  Thompson,  2011)  and  ask  any 
questions.  Additional  information  was  then  gathered  from  participants  such  as  age  and  experience 
(service  and  video  game).  Subsequently,  participants  took  part  in  the  experiment.  The  specific  testing 
procedure  was  determined  by  the  participant  group  to  which  each  of  the  participants  was  assigned.  Upon 
boarding  HMCS  Comer  Brook  for  the  first  time  during  this  experiment,  all  participants  received  a  safety 
briefing  from  a  qualified  submariner  regarding  the  use  of  the  EBS  and  the  EEBD. 

VCVS  Familiarization  and  Skill  Acquisition  Session  1-  Qualified  and  Experimental 
Groups  Only 

In  order  to  accommodate  work  schedules  and  availabilities,  Qualified  participants  used  the  VCVS  aboard 
HMCS  Comer  Brook.  However,  to  avoid  incidental  learning  and  to  provide  a  quiet  learning  environment, 
all  Experimental  participants  used  the  VCVS  in  Building  N60,  a  classroom  at  CFB  Esquimalt.  The 
Control  group  did  not  participate  in  the  VCVS  familiarization  and  skill  acquisition  session  as  these 
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participants  were  initially  trained  to  isolate  bulkhead  35  physically  aboard  HMCS  Comer  Brook  (see 
Section  0). 

At  the  beginning  of  the  VCVS  session,  the  laptop  keyboard  controls  and  functions  associated  with 
manoeuvring  the  participant’s  point  of  view  within  the  VCVS  were  identified  and  demonstrated  to  all 
participants.  Specifically,  the  experimenter  demonstrated  that  the  point  of  view  could  be  controlled  using 
the  mouse,  and  that  participants  could  manoeuvre  within  the  VCVS  using  either  the  standard  gaming 
“W/A/S/D”  keys  or  using  the  cursor  arrows  (up,  down,  left,  right).  Additionally,  participants  were 
informed  that  the  VCVS  had  a  built-in  map  function  that  provided  an  aerial  view  of  the  deck  that  they 
were  on  and  their  spatial  location  on  that  deck  within  the  VCVS.  An  information  sheet  explaining  these 
controls  and  functions  was  provided  to  all  participants  and  was  available  for  reference  whenever  required 
during  the  VCVS  session. 

Once  participants  were  sitting  in  front  of  the  laptop,  the  experimenter  loaded  the  VCVS  software  to  begin 
the  familiarization  trial.  In  the  familiarization  trial,  participants,  using  the  mouse  and  keyboard  controls, 
moved  from  the  Jetty  onto  the  Victoria  Class  submarine  and  down  the  main  access  hatch  guided  by  the 
experimenter.  Once  aboard,  participants  were  shown  how  to  isolate  bulkhead  35  within  the  simulated 
submarine  environment,  from  both  the  forward  and  aft  compartments.  The  experimenter  verbally  guided 
the  participants  to  each  valve  location  and  identified  the  name,  function  and  actions  that  were  required  in 
the  simulation  in  order  to  shut  the  valves  (i.e.,  click  to  turn  the  valve  or  locate  and  click  on  the  correct 
ratchet  to  apply  to  the  valve  when  required).  During  the  familiarization  trial,  the  compartment  (i.e., 
forward  or  aft)  that  the  participant  began  in  from  where  the  experimenter  guided  the  participant  through 
the  isolation  of  bulkhead  35  was  randomized  to  avoid  order  effects  on  learning.  Participants  had  the 
opportunity  to  ask  questions  throughout  the  familiarization  trial  in  the  VCVS  session. 

After  a  quick  break  the  experimenter  loaded  the  VCVS  software  to  begin  the  skill  acquisition  trials.  At  the 
beginning  of  each  trial,  one  of  six  different  starting  locations  (three  starting  locations  in  the  forward  and 
aft  compartments)  was  randomly  selected  by  the  software.  After  activating  the  bulkhead  35  isolation  task, 
the  experimenter  asked  the  participants  to  isolate  bulkhead  35  from  the  compartment  in  which  they  were 
currently  located.  At  the  beginning  of  each  trial  in  the  VCVS,  the  participants  were  presented  with  an 
instructional  prompt  that  listed  the  names  of  all  six  valves  that  were  to  be  shut  in  order  to  successfully 
isolate  bulkhead  35. 

The  participants  then  proceeded  to  interact  with  the  VCVS  in  the  following  ways: 

•  Participants  navigated  through  the  VCVS  using  the  mouse  and  keyboard  controls, 
correctly  locating,  identifying,  stating  the  function  of,  and  shutting  the  six  valves 
required  to  isolated  bulkhead  35; 

•  Upon  locating  each  valve,  the  participant  placed  the  crosshairs  (controlled  by  the 
computer  mouse)  on  the  valve.  If  this  valve  was  one  of  the  correct  valves  to  shut,  it 
would  turn  green.  To  simulate  shutting  the  valve,  the  participant  clicked  (once  with 
the  left  mouse  button)  on  the  valve  at  which  time  an  instructional  prompt  was 
presented  on  the  screen  that  identified  the  valve’s  name  and  function  with  a  green 
checkmark  indicating  the  valve  had  been  successfully  shut;  and, 

•  In  order  to  shut  a  valve  that  required  a  ratchet,  a  prompt  first  appeared  which 
instructed  the  participant  to  locate  the  ratchet.  In  order  to  locate  the  ratchet,  the 
participant  located  and  “looked”  at  the  correct  ratchet  in  its  storage  bracket  (using 
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the  mouse  to  position  the  crosshairs  over  it).  Once  the  crosshairs  were  positioned 
over  the  correct  ratchet,  the  ratchet  turned  green.  The  participant  then  clicked  (once 
with  the  left  mouse  button)  to  pick  up  the  ratchet  which  then  disappeared  from  view. 
The  participant  was  then  presented  with  another  instructional  prompt  to  utilize  the 
ratchet  to  shut  the  valve.  To  shut  the  valve,  the  participant  first  identified  the  correct 
valve  to  shut  (by  using  the  mouse  to  put  the  crosshairs  over  it).  Once  the  crosshairs 
were  placed  over  the  correct  valve,  which  turned  green  indicating  the  participant  had 
identified  the  correct  valve  to  shut,  the  participant  clicked  the  valve  (once  with  the  left 
mouse  button)  to  shut  it.  When  the  valve  was  clicked  the  ratchet  reappeared  in  the 
VCVS,  now  applied  on  the  valve,  simulating  the  ratchet  being  used  to  shut  the  valve. 
The  participant  then  received  the  instructional  prompt  that  identified  the  valve’s 
name  and  function  along  with  a  green  checkmark  which  indicated  the  valve  had 
been  successfully  shut.  The  last  instructional  prompt  that  appeared  to  the  participant 
informed  him  to  replace  the  ratchet.  In  order  to  replace  the  ratchet,  the  participant 
first  positioned  the  crosshairs  over  the  ratchet  currently  applied  to  the  valve,  which 
turned  green  indicating  this  was  the  ratchet  that  had  to  be  replaced,  and  then  the 
participant  clicked  the  ratchet  (once  with  the  left  mouse  button).  The  ratchet 
disappeared  off  of  the  valve  and  reappeared  immediately  in  the  storage  bracket, 
simulating  the  ratchet  being  stowed  after  use. 

Upon  successfully  isolating  bulkhead  35  in  the  VCVS,  Qualified  participants  completed  payment  forms, 
which  concluded  their  participation  in  the  study.  However,  upon  successfully  isolating  bulkhead  35  in  the 
VCVS,  Experimental  participants  were  given  directions  to  HMCS  Corner  Brook  for  their  second 
participation  session  before  being  dismissed  for  the  day. 

Submarine  Familiarization  Session  1  -  Control  Group  Only 

The  submarine  familiarization  session  began  with  the  experimenter  demonstrating  to  Control  participants 
how  to  isolate  bulkhead  35  aboard  HMCS  Comer  Brook  from  both  the  forward  and  aft  compartments. 
During  this  demonstration,  the  experimenter  guided  the  Control  participants  to  each  valve  location, 
identifying  the  valve’s  name,  function  and  the  action  required  to  shut  the  valve  (i.e.,  turn  the  valve  or 
apply  a  ratchet).  The  bulkhead  compartment  (forward  or  aft)  from  which  Control  participants  were 
initially  shown  how  to  isolate  bulkhead  35  was  randomized  by  the  experimenter  to  avoid  order  effects  as 
in  the  VCVS  training  and  familiarization  session  for  the  Experimental  group.  During  the  submarine 
familiarization  session,  Control  participants  had  the  opportunity  to  ask  questions  but  were  not  given  any 
additional  opportunities  to  practise  aboard.  Upon  completion  of  the  submarine  familiarization  trial, 
participants  were  escorted  off  the  boat;  this  concluded  their  participation  in  session  1. 

Submarine  Skill  Acquisition  Session  2  -  Control  and  Experiment  Groups  Only 

Two  days  following  the  familiarization  session  (either  in  the  VCVS  or  aboard),  Control  and  Experimental 
participants  participated  in  the  submarine  skill  acquisition  session  aboard  HMCS  Comer  Brook.  At  this 
point  participants  were  lead  by  the  experimenter  to  one  of  six  randomized  starting  locations  (three  in  the 
forward  and  three  in  the  aft  compartment).  Participants  were  then  asked  to  isolate  bulkhead  35  from  the 
compartment  in  which  they  currently  were  located.  Again,  in  order  for  a  participant  to  complete  the 
submarine  skill  acquisition  session,  they  had  to  correctly  locate,  identify,  explain  the  function  and  the 
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action  required  to  shut  all  six  valves  associated  with  isolating  bulkhead  35  perfectly  one  time  from  either 
the  forward  or  aft  compartment.  When  they  had  successfully  demonstrated  the  isolation  of  bulkhead  35, 
they  completed  a  payment  form  and  were  escorted  by  the  experimenter  off  HMCS  Comer  Brook, 
concluding  their  participation  in  the  study. 
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4.  Results 

VCVS  Utilization 

The  following  sub-sections  describe  how  Experimental  participants  utilized  and  interacted  with  the  VCVS 
while  learning  how  to  isolate  bulkhead  35. 

Observed  Learning  Strategies 

During  the  VCVS  skill  acquisition  trials,  Experimental  participants  were  observed  to  engage  in  two 
distinct  learning  strategies  while  they  trained  on  the  VCVS.  These  two  learning  strategies  emerged  as  the 
result  of  allowing  participants  to  interact  with  and  utilize  the  VCVS  in  whatever  way  they  thought  would 
benefit  their  ability  to  learn  how  to  isolate  bulkhead  35  on  the  Victoria  Class  submarine.  The  two 
strategies  Experimental  participants  demonstrated  when  required  to  identify  valve  names  and  functions 
were: 

•  Reading  the  valve’s  name  and  corresponding  function  directly  off  the  instructional 
prompt  that  was  presented  immediately  after  clicking  on  the  valve;  and, 

•  Generating  an  educated  guess  of  the  valve’s  name  and  function  prior  to  clicking  on 
the  valve  and  then  verifying  their  response  with  the  correct  answer  provided. 

VCVS  Map  View  Activations 

To  determine  how  often  the  “Map  View”  function  was  utilized  by  both  Qualified  and  Experimental 
participants,  the  VCVS  log  files  were  reviewed.  Only  the  activations  of  the  “Map  View”  function  that 
were  initiated  by  the  participants  during  VCVS  skill  acquisition  trials  were  included  in  the  analysis  (i.e., 
map  view  activations  initiated  by  the  experimenter  to  demonstrate  the  existence  of  the  feature  were  not 
counted)  to  understand  the  frequency  with  which  the  “Map  View”  was  utilized  by  Qualified  and 
Experimental  participants.  Results  indicate  that  on  average,  Qualified  participants  activated  the  “Map 
View”  function  1.1  times  during  their  participation,  whereas  the  “Map  View”  function  was  never 
activated  by  the  Experimental  participants. 

Strengths  of  the  VCVS 

The  following  sub-section  describes  the  strengths  associated  with  the  VCVS. 

Point  of  View  Movement  Controls 

During  the  VCVS  skill  acquisition  session,  the  keyboard  control  preferences  associated  with  manoeuvring 
within  the  VCVS  for  each  of  the  Qualified  and  Experimental  participants  were  recorded  and  analyzed. 

The  results  showed  that  40%  and  60%  of  the  Qualified  and  Experimental  participants  respectively,  used 
the  standard  gaming  letters  “W/A/S/D”.  The  remaining  percentage  of  participants  in  each  testing  group 
used  the  cursor  arrows  (up,  down,  left,  right)  to  manoeuvre  within  the  VCVS.  The  two  different  methods 
of  control  demonstrate  the  VCVS’s  ability  to  accommodate  different  users’  preferences  (i.e.,  allowing 
users  to  use  preferred  keyboard  controls). 
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Additionally,  observations  throughout  the  VC  VS  sessions  led  to  conclusions  that  Qualified  and 
Experimental  participants  were  able  to  easily  ascend  and  descend  between  the  different  deck  levels  using 
the  ladders,  as  well  as  manoeuvre  through  bulkheads.  The  ease  with  which  participants  could  move 
throughout  the  simulated  environment  could  possibly  be  attributed  to  the  automatic  instructional  flags 
presented  within  VCVS  which  identified  required  actions  and  their  associated  keyboard  letter  needed  to 
correctly  perform  the  action. 

Qualitative  Feedback 

The  qualitative  feedback  provided  by  the  Experimental  participants  was  analyzed  and  categorized  to 
reflect  general  themes  associated  with  using  the  VCVS  to  learn  how  to  isolate  bulkhead  35. 

Comments  Provided  During  and  After  VCVS  Familiarization  and  Skill  Acquisition 
Session 

During  the  VCVS  familiarization  and  skill  acquisition  session  several  Experimental  participants  indicated 
that  they  were  frustrated  by  the  inability  to  read  any  information  off  the  valve  VV803  environmental  label 
located  in  the  Senior  Ratings  Mess  and  inability  to  read  parts  of  the  information  provided  on  valve 
VV608  environmental  label  within  the  VCVS.  Since  participants  were  told  that  they  could  use  the 
environmental  labels  to  identify  the  valve  names  and  functions  (as  they  would  be  able  to  do  aboard  the 
submarine),  they  were  observed  to  be  frustrated  and  disappointed. 

Upon  conclusion  of  the  VCVS  familiarization  and  skill  acquisition  session,  several  Experimental 
participants  were  asked  to  provide  general  comments  about  their  experiences  training  and  interacting  with 
the  VCVS.  The  majority  of  the  Experimental  participants  commented  that  the  graphics  incorporated 
within  the  VCVS  were  quite  good.  Some  of  the  participants  remarked  that  the  simulation  had  some  of  the 
best  graphics  that  they  had  ever  seen  and/or  interacted  with  when  compared  to  other  video  games  on  the 
market.  Some  participants,  however,  commented  on  how  they  found  it  difficult  to  understand  and  recall 
the  valve  names  and  functions  in  English  because  French  was  their  first  language.  Consequently,  these 
participants  explained  that  they  felt  their  poor  procedural  performance  was  a  result  of  this  language 
bander. 

Comments  Provided  After  Submarine  Skill  Acquisition  Session 

During  the  submarine  skill  acquisition  session,  one  Experimental  participant  struggled  to  locate  the 
ratchet  required  to  close  LPB  803.  This  participant  identified  that  the  reason  he  was  struggling  was 
because  it  was  not  located  in  the  cabinet  where  he  was  trained  to  locate  it  within  the  VCVS.  Ele 
commented  that  the  ratchet  in  the  simulation  was  located  incorrectly  (attached  to  the  wall  within  the 
cabinet)  compared  to  its  real  location  aboard  (attached  to  the  inside  of  the  cabinet  door).  This  suggests 
that  the  participant  had  developed  an  understanding  of  the  spatial  location  associated  with  the  LPB  803 
ratchet  through  his  use  of  the  VCVS. 

After  being  physically  aboard  E1MCS  Comer  Brook  and  having  completed  their  participation  in  the 
VCVS  study,  several  Experimental  participants  were  asked  or  volunteered  their  thoughts  about  the  VCVS 
and  how  realistic  they  felt  the  VCVS  represented  the  actual  submarine.  Several  participants  commented 
that  they  believed  the  simulation  was  very  realistic  and  representative  of  the  true  submarine  environment. 
Some  participants  commented  that  they  felt  as  though  they  had  been  quite  prepared  for  the  submarine 
skill  acquisition  session,  specifically  accrediting  the  VCVS  with  providing  good  spatial  training  of  the 
various  areas  and  levels  within  the  submarine. 
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5.  Technical  Limitations  And  recommendations  associated  with  the  VCVS 

This  section  identifies  the  technical  limitations  associated  with  the  VCVS  that  were  observed  during  the  Experimental  group’s  VCVS 
Familiarization  and  Skill  Acquisition  Session  1.  Table  5-1  below  summarizes  the  limitations  that  Experimental  participants  faced  while  interacting 
with  and  utilizing  the  VCVS. 

Table  5-1:  Technical  Limitations  and  Proposed  Recommendations  Associated  with  the  VCVS. 


VCVS  Technical  Limitations  and  Proposed  Recommendations 

# 

Technical  Limitations 

Recommendations 

1.0 

During  some  of  the  testing  sessions,  the  VCVS  screen  periodically 
froze.  The  frame  delays  decreased  the  overall  fluidity  of  visual 
movements  and  had  participants  questioning  the  accuracy  of  their 
movement  and  actions/  inputs  into  the  simulation. 

Minimize  VCVS  frame  delays  to  increase  the  fluidity  and  realism 
associated  with  the  VCVS. 

2.0 

Participants  could  not  read  the  valve  name  of  VV608  and  LPB  803  or 
the  valve  name  and  function  of  VV803  off  of  the  environmental  labels 
simulated  within  the  VCVS.  When  this  label  could  not  be  read  in  the 
VCVS  as  they  could  be  aboard  the  real  submarine,  participants  were 
observed  to  be  frustrated.  Experiencing  the  VCVS’s  inability  to 
replicate  an  action  that  could  be  performed  in  real  life,  may  reduce  the 
realism  and  thus,  overall  credibility  associated  with  the  VCVS. 

Allowing  the  user  to  move  close  enough  to  the  environmental  labels 
within  the  VCVS  to  read  all  valve  labels,  will  increase  not  only  the 
realism  and  credibility  of  the  VCVS,  but  may  also  improve  the 
overall  effectiveness  of  the  VCVS  to  procedurally  train  the 
bulkhead  35  isolation  task. 

3.0 

Participants  were  observed  to  struggle  more  with  shutting  valves  that 
required  the  use  of  a  ratchet.  As  more  and  more  participants  were 
observed,  it  became  clear  that  the  controls  required  to  shut  a  valve 
with  a  ratchet  were  non-intuitive. 

*See  section  5.1  below  for  a  detailed  recommendation  regarding  this 
issue. 

4.0 

Participants  received  a  “Wrong  Action”  message  when  they  clicked 
on  the  green  coloured  ratchet  required  to  shut  VV608.  Participants 
were  observed  to  be  confused  because  they  were  clicking  on  an 
identified  ratchet,  presented  in  green,  which  they  were  trained  to 
understand  as  an  indicator  of  a  correct  ratchet  selection.  It  was 

When  the  user  can  turn  a  valve  green  (by  using  a  computer  mouse  to 
position  the  crosshairs  over  the  correct  valve),  the  user  (at  the  exact 
same  viewing  distance)  should  be  able  to  shut  the  valve. 
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VCVS  Technical  Limitations  and  Proposed  Recommendations 

# 

Technical  Limitations 

Recommendations 

discovered  that  if  the  participant  moved  closer  to  the  ratchet 
associated  with  VV608  and  clicked  again  on  the  ratchet,  the  “Wrong 
Action”  message  disappeared  and  the  typical  instructional  prompts 
associated  with  isolating  VV608  were  presented  on  the  screen. 

5.0 

A  participant  indicated  that  he  was  struggling  to  find  the  LPB  803 
ratchet  in  the  cabinet  located  in  the  Commanding  Officer’s  cabin. 

Upon  comparing  the  ratchet  location  in  the  VCVS  to  its  location 
aboard  HMCS  Comer  Brook,  the  reason  this  participant  struggled  to 
find  the  ratchet  was  because  the  ratchet  in  the  simulation  trained  the 
participant  to  look  in  the  cabinet  when  in  fact  aboard  HMCS  Comer 
Brook,  the  ratchet  was  attached  to  the  cabinet  door. 

Ensure  that  the  LBP  803  ratchet  is  positioned  in  the  VCVS  as  it  is 
located  aboard  HMCS  Corner  Brook. 

6.0 

When  a  participant  was  in  front  of  the  ascending  ladder  at  bulkhead 

35  and  pressed  the  “R”  letter  as  prompted  to  use  ladder,  the 
participant  descended  down  the  adjacent  ladder  to  the  Mast  Well. 
Participants  were  observed  to  be  very  confused  as  they  anticipated 
they  would  take  the  ascending  ladder  up  to  the  control  room. 

If  a  user  cannot  use  the  ladder  at  bulkhead  35  to  ascend  to  the 
control  room,  the  instructional  prompt  (i.e.,  informing  the  user  to 
press  the  letter  “R”  to  use  ladder)  should  be  removed  from  the 

VCVS.  The  only  way  a  user  should  be  able  to  descend  down  to  the 
Mast  Well  compartment  is  by  positioning  himself  in  front  of  the 
descending  ladder  (leading  to  the  Mast  Well)  at  bulkhead  35  and 
correctly  following  the  instructional  prompt. 

7.0 

In  the  VCVS,  the  door  leading  to  the  Sonar  Cabinet  Space  was  open, 
where  aboard  HMCS  Comer  Brook  the  door  was  found  to  be  closed 
unless  personnel  were  working  within  that  area.  As  participants  were 
not  required  to  open  the  door  in  the  VCVS  this  may  have  caused 
confusion  for  several  participants  who  struggled  with  locating  the 
VV803  valve  in  the  Sonar  Cabinet  Space  aboard  HMCS  Comer 

Brook. 

Aboard  HMCS  Comer  Brook,  the  Sonar  Cabinet  Space  door  was 
always  shut  unless  personnel  were  working  within  this  compartment. 
At  the  time  of  testing,  it  had  not  been  determined  if  having  the  Sonar 
Cabinet  Space  door  shut  was  a  Victoria  Class  submarine  standard  or 
crew  standard.  Therefore,  before  the  VCVS  is  redesigned  (requiring 
user  to  open  the  door)  further  information  upon  this  standard  should 
be  obtained  to  improve  consistency  between  HMCS  Comer  Brook 
and  the  VCVS. 

8.0 

A  few  francophone  participants  were  observed  struggling  and 

Valve  functions  should  be  presented  in  both  English  and  French. 
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VCVS  Technical  Limitations  and  Proposed  Recommendations 

# 

Technical  Limitations 

Recommendations 

becoming  frustrated  with  recalling  valve  functions.  More  than  one 
francophone  participant  commented  that  he  had  a  difficult  time 
understanding  the  functional  meaning  associated  with  the  valves. 

Incorporating  both  languages  within  the  VCVS  will  provide  a  better 
learning  environment  for  francophone  users. 
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VCVS  Recommendations  for  Technical  Limitation  3.0 

A  Hierarchical  Task  Analysis  (HTA)2  was  performed  on  shutting  a  bulkhead  35  isolation  valve  requiring 
a  ratchet  in  real  life  aboard  the  submarine  to  understand  how  this  task  was  executed  within  the  VCVS. 
Table  5-2  presents  the  HTA  results  of  the  shutting  bulkhead  35  valve  requiring  a  ratchet. 

Table  5-2:  Tabular  HTA  for  Shutting  Bulkhead  35  Valve  Requiring  a  Ratchet. 


HTA  for  Shutting  a  Bulkhead  35  Isolation  Valve  Requiring  Ratchet 


0.  Shut  off  bulkhead  35  isolation  valve  with  ratchet. 
Plan  0:  Do  1  then  2  then  3  then  4  then  5. 


1.0  Retrieve  correct  ratchet. 

Plan  1:  Do  1.1  then  1.2  then  1.3. 


1 . 1  Locate  correct  ratchet  in  storage  bracket. 

Subtasks 

1 .2  Remove  ratchet  from  storage  bracket. 

1.3  Transport  ratchet  to  valve  area. 

2.0  Apply  ratchet  to  valve. 

Plan  2:  Do  2. 1  then  2.2. 

Subtasks 

2.1  Locate  correct  valve. 

2.2  Slide  ratchet  head  onto  valve  axel. 

3.0  Shut  valve. 

Plan  3:  Do  3.1. 

Subtasks 

3.1  Use  ratchet  to  turn  valve  axel. 

4.0  Remove  ratchet  off  valve. 

Plan  4:  Do  4.1. 

Subtasks 

4.1  Slide  ratchet  head  off  valve  axel. 

5.0  Stow  ratchet  in  storage  bracket. 

Plan  5.0:  Do  5.1  then  5.2  then  5.3. 

5.1  Transport  ratchet  to  storage  bracket  area. 

Subtasks 

5.2  Locate  correct  ratchet  storage  bracket. 

5.3  Place  ratchet  in  storage  bracket. 

Upon  reviewing  the  HTA  and  the  results  of  the  current  experiment,  it  is  suggested  that  Task  3.0  be 
maintained  but  that  the  sub-tasks  associated  with  both  Task  1.0  and  Task  2.0  be  combined  into  a  single 


1  The  HTA  was  first  developed  by  John  Annett  and  Keith  Duncan,  as  cited  in  Shepherd  and  Stammers,  2005,  p.140. 
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task.  Also,  the  sub-tasks  associated  with  Task  4.0  and  Task  5.0  should  be  combined  into  a  single  task. 
Consequently,  the  following  three  tasks  would  be  simulated  within  VC  VS: 

•  VCVS  Task  1  Retrieve  and  apply  ratchet  to  valve; 

•  VCVS  Task  2  Shut  valve;  and, 

•  VCVS  Task  3  Remove  and  stow  ratchet. 

Using  these  three  tasks  as  a  basis,  the  following  recommendations  can  be  made  to  increase  the  intuitive 
nature  of  this  simulated  task  within  the  VCVS. 

VCVS  Task  1:  Retrieve  and  apply  ratchet  to  valve 

The  first  recommendation  is  to  have  the  instructional  prompt  instruct  the  user  to  locate  and  apply  the 
ratchet,  clearly  explaining  the  task  that  should  be  executed.  In  working  with  the  VCVS,  the  study  team 
researchers  had  discussed  the  possibility  of  implementing  a  click  and  drag  functionality  to  choose, 
transport  and  apply  the  ratchet  in  the  VCVS.  Upon  greater  analysis  of  the  task,  it  is  recommended  that 
implementing  a  ‘click  and  drag’  functionality  to  execute  this  task  within  the  VCVS  would  provide  a  more 
realistic  representation  of  the  actual  physical  task.  It  is  suggested  that  this  task  be  designed  for  execution 
as  described  in  Table  5-3. 

Table  5-3:  Proposed  User  Actions  to  Execute  VCVS  Task  1. 


VCVS  Task  1:  Retrieve  and 
Apply  Ratchet  to  Valve 

Proposed  User  Actions 

1.1  Locate  correct  ratchet  in 
storage  bracket. 

Using  computer  mouse,  user  positions  the  crosshairs  over  the  correct 
ratchet  residing  in  the  storage  bracket. 

*Ratchet  is  visible  to  user  and  resides  in  storage  bracket. 

1.2  Remove  ratchet  from 
storage  bracket. 

With  the  crosshairs  positioned  over  ratchet,  user  depresses  left  click 
button  on  computer  mouse. 

*Ratchet  remains  visible  to  user  in  the  storage  bracket. 

1.3  Transport  ratchet  to  valve 
area. 

User  maintains  left  click  button  depression  and  drags  ratchet  from 
storage  bracket  to  correct  valve  area. 

*Ratchet  remains  visible  to  user  during  dragging  action. 

1.4  Locate  correct  valve. 

User  positions  ratchet  on  correct  valve. 

*Ratchet  remains  visible  to  user  residing  on  valve. 

1.5  Slide  ratchet  head  onto 
valve  axel. 

Once  ratchet  is  on  correct  valve,  user  releases  left  click  button,  securing 
ratchet  onto  the  valve. 

*Ratchet  remains  visible  to  user  residing  on  valve. 
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VCVS  Task  2:  Shut  Valve 

Since  Task  2.0  (Shut  valve)  is  a  distinct  task,  it  is  suggested  that  in  order  to  shut  the  valve,  the  user  should 
make  a  separate  left  click  on  the  ratchet  that  they  just  applied  to  the  valve  (as  described  in  Table  5-4). 
When  the  single  left  click  is  performed,  the  instructional  prompt  identifying  the  valve  name  and  function 
should  be  presented  and  the  ratchet  should  also  provide  a  visual  indication  to  the  user  that  they  have 
successfully  shut  the  valve. 

Table  5-4:  Proposed  User  Actions  to  Execute  VCVS  Task  2. 


VCVS  Task  2:  Shut  Valve 

Proposed  User  Actions 

2. 1  Use  ratchet  to  turn  valve 
axel. 

Using  computer  mouse,  user  positions  the  crosshairs  over  the  correct 
ratchet  residing  on  the  valve. 

With  the  crosshairs  positioned  over  ratchet,  user  depresses  and  releases 
left  click  button  on  computer  mouse. 

*Ratchet  remains  visible  to  user  while  making  a  360  degree  revolution. 

VCVS  Task  3:  Remove  and  Stow  Ratchet 

In  order  to  maintain  functional  consistency  within  this  task,  it  is  recommended  that  actions  required  of  the 
user  to  remove  and  stow  the  ratchet  be  performed  in  a  manner  that  is  identical  to  proposed  VCVS  Task  1 
above.  It  is  suggested  that  VCVS  Task  3  be  designed  within  the  simulation  to  be  executed  according  to 
Table  5-5. 


Table  5-5:  Proposed  User  Actions  to  Execute  VCVS  Task  3. 


VCVS  Task  3:  Remove  and 
Stow  Ratchet 

Proposed  User  Actions 

3. 1  Slide  ratchet  head  off  valve 
axel. 

Using  computer  mouse,  user  positions  the  crosshairs  over  the  ratchet 
residing  on  the  valve. 

With  the  crosshairs  positioned  over  ratchet,  user  depresses  left  click 
button  on  computer  mouse. 

*  Ratchet  remains  visible  to  user  residing  on  the  valve. 

3.2  Transport  ratchet  to  storage 
bracket  area. 

User  maintains  left  click  button  depression  and  drags  ratchet  from 
valve  to  storage  bracket  area. 

*  Ratchet  remains  visible  to  user  during  dragging  action. 

3.3  Locate  correct  ratchet  storage 
bracket. 

User  positions  ratchet  on  correct  storage  bracket. 

*  Ratchet  remains  visible  to  user,  hovering  over  storage  bracket. 

3.4  Place  ratchet  in  storage 
bracket. 

Once  ratchet  is  on  correct  storage  bracket,  user  release  left  click 
button,  securing  ratchet  in  storage  bracket. 
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VCVS  Task  3:  Remove  and 
Stow  Ratchet 

Proposed  User  Actions 

*  Ratchet  remains  visible  to  user,  residing  in  storage  bracket. 
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6.  CAE  PS  Observations  OF  VCVS 

In  addition  to  the  recommendations  put  forth  in  Section  5,  DRDC  Toronto  requested  that  I  (C.  Kersten) 
provide  additional  thoughts  and  recommendations  resulting  from  personal  experiences  interacting  with 
and  utilizing  the  VCVS  as  a  training  tool  in  preparation  for  this  study.  The  following  sub-sections  identify 
recommendations  that  I  believe  may  enhance  the  realism  of  the  VC  VS  and  to  improve  its  overall 
effectiveness  as  a  training  tool. 

Spatial  Training 

I  believe  that  the  VCVS  is  currently  sophisticated  enough  to  provide  general  spatial  knowledge  of  the 
Victoria  Class  submarine.  Evidence  to  support  this  belief  arises  from  observing  Experimental  participants 
confidently  perform  correct  spatial  manoeuvres  when  isolating  bulkhead  35  aboard  HMCS  Comer  Brook. 
In  addition,  two  Experimental  participants  were  able  to  perform  the  bulkhead  35  isolation  task  perfectly 
during  their  first  attempt  aboard  HMCS  Corner  Brook.  In  fact,  one  of  these  two  Experimental  participants 
was  so  eager  to  demonstrate  the  bulkhead  35  isolation  task  that  he  had  to  be  asked  to  slow  down  (i.e.,  not 
walk/race  so  quickly  ahead  of  me)  so  that  I  could  observe  and  record  his  performance.  Being  able  to 
perform  the  task  perfectly,  thereby  executing  both  spatial  and  procedural  knowledge  correctly  during  the 
first  time  aboard  HMCS  Comer  Brook  is  a  persuasive  indication  as  to  the  VCVS’s  ability  to  effectively 
train  personnel  on  the  spatial  layout  of  the  Victoria  Class  submarine. 

Additionally,  my  own  personal  experiences  training  with  the  VCVS  in  preparation  for  conducting  the 
study  strengthens  the  belief  that  the  VCVS  can  effectively  train  the  spatial  layout  of  the  Victoria  Class 
submarine.  Upon  boarding  HMCS  Comer  Brook  for  the  first  time,  with  only  previous  exposure  to  the 
submarine’s  spatial  layout  through  VCVS  training,  I  was  able  to  not  only  recognize  and  correctly  identify 
compartments  within  the  submarine,  but  also  able  to  manoeuvre  correctly  to  the  general  valve  locations. 
Having  such  a  great  understanding  of  the  spatial  layout  of  HMCS  Comer  Brook  when  being  only  aboard 
for  the  first  time,  is  a  huge  testament  to  the  spatial  training  abilities  of  the  VCVS. 

However,  the  only  recommendation  that  I  could  make  to  enhance  the  spatial  training  effectiveness  of  the 
VCVS  would  be  to  represent  the  valves  more  realistically  within  their  complex  and  camouflaged 
environments.  When  aboard  HMCS  Comer  Brook  for  the  first  time,  I  knew  generally  where  to  expect  the 
valves  to  be  located  within  the  space,  but  had  a  little  bit  of  difficulty  locating  valves  VV801  and  LPB  803. 
Specifically,  I  experienced  that  the  immediate  environments  surrounding  valves  VV801  and  LPB  803  (in 
both  forward  and  aft  compartments)  were  much  more  complex  than  simulated  within  the  VCVS. 

Procedural  Training 

Additionally,  I  believe  that  the  following  recommendations  may  improve  the  VCVS’s  ability  to  train  the 
procedural  knowledge  associated  with  isolating  bulkhead  35. 

Generation  of  Valve  Locations,  Names  and  Functions 

By  clicking  on  a  green  coloured  valve  within  the  VCVS,  the  valve’s  name  and  function  automatically 
appear.  While  this  is  an  excellent  way  to  first  introduce  each  valve  (green  colour  identifies  valve  and 
prompt  explains  name  and  function),  the  VCVS  software  should  be  designed  to  have  subsequent  levels  of 
difficulty  to  challenge  and  appeal  to  the  more  experienced  personnel.  Requiring  the  user  to  locate, 
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identify  and  explain  valve  functions  on  their  own  first  before  providing  feedback,  will  increase  the  level 
of  difficulty  within  the  VCVS  and  provide  a  more  complex  training  experience.  Therefore,  the  current 
version  could  be  considered  the  “Beginner”  level  to  ensure  that  non-experienced  users  are  introduced  but 
not  overwhelmed  by  the  task.  Whereas,  a  more  “Senior”  level  version  of  the  VCVS  could  be  activated  to 
challenge  the  more  experienced  users  (i.e.,  provide  a  more  difficult  testing  environment  to  improve 
memory). 

Label  Discrepancy 

During  the  study,  a  discrepancy  arose  between  valve  VV607  and  VV606  functional  responses  provided 
by  Qualified  participants  to  the  functions  presented  in  the  VCVS.  Review  of  the  forward  compartment 
environmental  label,  which  lists  each  valve  and  function,  identifies  valves  VV607  and  VV606  functions 
to  be  associated  with  the  Bathroom,  matching  the  “Heads”  responses  provided  by  the  Qualified 
participants.  This  discrepancy  indicates  that  the  VCVS  valve  functions  need  to  be  revised.  NeLCoE 
should  conduct  a  focus  group  with  submariner  trainers  to  determine  acceptable  valve  functional  meanings 
to  be  incorporated  within  the  VCVS. 

I  additionally  recognized  that  the  valve  environment  labels  did  not  match  functions  presented  in  the 
simulation.  In  the  VCVS  an  association/  link  between  the  valve  environmental  label  to  that  of  the 
accepted  functional  meaning  should  be  presented  to  the  user.  For  example,  the  valve  environmental  label 
identifies  valve  VV607  to  be  “A.T.U.  2  BHD  Isolating  Valve”.  Whereas,  in  the  VCVS  the  functional 
meaning  associated  with  VV607  is  “Ventilation  Supply  Isolating  Shut”.  From  experience  working  with 
qualified  submariners,  I  learned  that  the  A.T.U.  stands  for  “Air  Treatment  Unit”  which  means  that  it  is  a 
“Ventilation”  valve.  By  explaining  what  the  valve  label  acronym  represents,  the  user  can  make  the 
appropriate  association  between  what  is  presented  on  the  valve  environmental  label  and  its  associated 
function. 

Valve  Direction 

During  the  experimental  team’s  familiarization/  qualification  day  aboard  HMCS  Corner  Brook,  qualified 
submariners  identified  the  importance  of  understanding  which  direction  a  bulkhead  35  isolating  valve  has 
to  be  turned  in  order  to  be  shut.  More  than  one  qualified  submariner  also  indicated  that  one  particular 
valve  (VV803)  required  a  different  directional  turn  depending  upon  the  compartment  (forward  or  aft)  in 
which  it  was  being  shut.  Since  directional  training  was  not  incorporated  within  the  VCVS,  the  valve’s 
shut  direction  was  not  tested  during  this  study.  However,  these  findings  indicated  that  the  direction  of  the 
turn  is  an  important  requirement  in  learning  how  to  isolate  bulkhead  35,  which  suggests  that  NeFCoE 
consider  incoiporating  the  directional  turn  associated  with  shutting  each  valve  into  the  VCVS. 
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7.  Strengths  and  Limitations  of  the  Study 

The  following  sub-sections  identify  the  strengths  and  limitations  associated  with  the  VC  VS  study. 

Strengths  of  the  study 

The  experimenters  were  able  to  realistically  represent  the  training  protocol  normally  followed  aboard  the 
submarine  by  acquiring  an  understanding  of  the  procedure  through  training  with  various  qualified 
submariners  aboard  the  submarine.  In  fact,  the  major  strength  associated  with  this  study,  was  the  ability  to 
realistically  replicate  Victoria  Class  submarine  bulkhead  35  isolation  training  aboard  HMCS  Comer 
Brook  and  within  the  VC  VS  simulated  environment. 

Limitations  of  the  study 

Scientific  studies  are  not  immune  to  limitations.  Upon  reflection,  the  following  limitations  were  identified 
as  potentially  influencing  the  study’s  findings. 

Unavoidable  Submarine  Incidental  Spatial  Learning 

During  the  submarine  skill  acquisition  sessions  (involving  Control  and  Experimental  participants), 
opportunities  arose  for  participants  to  receive  spatial  training  that  did  not  occur  with  the  VCVS.  The  first 
opportunity  where  participants  may  have  potentially  gained  additional  spatial  awareness  of  HMCS  Comer 
Brook  was  when  they  participated  in  required  safety  briefings.  The  safety  briefings  were  performed 
aboard  HMCS  Comer  Brook  at  a  convenient  location  which  was  determined  by  the  qualified  submariner 
who  conducted  the  briefing.  This  provided  the  participant  an  opportunity  to  acquire  spatial  knowledge 
while  transiting  to  the  safety  briefing  location. 

Additional  and  completely  unavoidable  opportunities  for  incidental  spatial  learning  occurred  between 
testing  trials  as  the  participants  moved  from  the  ending  location  of  one  trial  to  the  starting  location  of  the 
next  trial.  Since  starting  locations  were  randomized,  the  distance  traveled  and  thus  the  amount  of  spatial 
exposure  to  the  submarine  varied  between  trials.  Thus,  there  were  two  opportunities  for  incidental  spatial 
learning,  which  varied  inconsistently,  during  testing  aboard  the  submarine  that  were  not  afforded  to 
participants  during  testing  with  the  VCVS. 

Testing  Criteria 

In  order  to  be  qualified  to  isolate  bulkhead  35,  a  trainee  is  required  to  demonstrate  perfect  performance 
one  time  from  either  a  forward  or  aft  approach  to  the  bulkhead.  In  this  study  the  participant  had  to  locate, 
identify,  describe  the  functions  and  actions  associated  with  shutting  all  six  valves  without  error  or 
assistance  from  either  a  forward  or  aft  compartment  in  order  to  achieve  criterion.  Consequently,  no 
participants  had  to  demonstrate  perfect  proficiency  from  both  directions.  Review  of  the  starting  locations 
that  were  randomly  picked  for  the  submarine  skill  acquisition  testing  sessions,  revealed  that  some 
participants  were  only  tested  from  one  direction  (i.e.,  demonstrate  bulkhead  35  task  only  from  a  forward 
or  aft  approach).  It  could  be  suggested  that  these  participants  may  have  been  at  a  slight  advantage, 
repetitively  tracing  their  spatial  manoeuvres,  never  having  to  cross  bulkhead  35  and  reorient  themselves 
with  the  mirrored  valve  locations. 
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Additionally,  in  comparing  bulkhead  35  task  from  both  compartments  it  was  acknowledged  that  there 
were  differences  between  the  spatial  location  of  the  valve  (i.e.,  aft  of  bulkhead  35  LPB  803  is  hidden 
within  a  cabinet  whereas  forward  of  bulkhead  35  LPB  803  is  exposed)  and  the  method  required  for  it  to 
be  shut  (i.e.,  aft  of  bulkhead  35  LPB  803  requires  a  ratchet  to  shut,  whereas  forward  of  bulkhead  35  LPB 
803  is  a  hand  turn  valve).  Based  upon  these  differences,  it  could  be  argued  that  it  may  be  more  difficult  to 
perform  this  task  from  one  compartment  in  comparison  to  the  other.  Again,  because  the  starting  location 
in  which  a  participant  had  to  isolate  bulkhead  35  was  randomized,  some  participants  may  have  been 
required  to  isolate  bulkhead  35  from  the  easier  compartment. 

Previous  Submarine  Exposure 

As  shown  in  Table  3-3  the  Experimental  and  Control  participants  had  little  experience  aboard  submarines 
when  compared  to  Qualified  participants.  One  of  the  Experimental  participants  (20  minutes  of  exposure) 
and  two  of  the  Control  participants  (65  minutes  and  1 6  hours  of  exposure)  had  some  previous  submarine 
exposure.  It  appears  that  their  previous  exposure  to  submarines  had  limited,  if  any,  influence  on  their  task 
performance.  Comparing  the  submarine  skill  acquisition  testing  session  results  of  these  three  participants 
to  the  participants  within  their  group  revealed  that  these  three  participants  were  not  the  most  successful 
(i.e.,  did  not  achieve  task  performance  criterion  in  fewest  trials  nor  commit  the  fewest  spatial/path  errors). 
This  finding  suggests  that  a  small  amount  of  prior  exposure  to  the  submarine  did  not  impact  the  overall 
results  of  the  study. 

Experimental  Participants 

As  identified  in  Sections  5  and  6,  the  VCVS  is  not  without  its  limitations.  Observation  of  the 
Experimental  participants  while  they  practiced  with  the  VCVS  identified  the  following  factors  as 
potential  influences  on  task  performance. 

Inability  to  Verify  Procedural  Learning 

Since  the  VCVS  and  the  study  criterion  did  not  restrict  Experimental  participants  from  reading  valve 
names  and  functions  directly  off  the  screen  during  the  VCVS  familiarization  and  skill  acquisition 
sessions,  it  is  quite  possible  that  Experimental  participants  had  learned  spatially  but  not  procedurally 
(valve  names  and  associated  function)  how  to  isolate  bulkhead  35  prior  to  their  submarine  skill 
acquisition  session.  Since  the  aids  were  available  during  the  VCVS  sessions,  it  is  not  possible  to  know  if  a 
participant  would  have  achieved  task  criterion  without  benefit  of  this  aid  (i.e.,  by  relying  on  memory  for 
the  procedures).  The  evidence  suggests  that  the  Experimental  participants  may  have  not  been  as 
procedurally  prepared  for  the  submarine  skill  acquisition  testing  session  as  anticipated. 

Legibility  and  Discrepancies  with  Valve  Labels 

The  inability  to  read  certain  valve  environmental  labels  within  the  VCVS  and  the  discrepancies  that  arose 
between  the  real  submarine  environmental  labels  and  functions  presented  within  the  VCVS,  were  two 
factors  that  could  have  affected  the  performance  of  the  Experimental  participants  during  the  submarine 
skill  acquisition  session.  It  could  be  hypothesized  that  since  Control  participants  were  able  to  clearly  read 
all  valve  environmental  labels  aboard  the  submarine,  they  were  able  to  make  direct  functional 
associations  between  what  the  environmental  label  read  and  what  they  had  to  functionally  verbalize  in 
order  to  successfully  complete  the  isolation  task.  Whereas,  when  Experimental  participants  could  not  read 
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the  valve  environmental  labels  within  the  VCVS,  they  were  unable  to  build  strong  functional  associations, 
making  it  difficult  to  recall  valve  functions  when  reading  valve  environment  labels  aboard  the  submarine. 

Language  Barrier 

The  last  factor  that  could  have  influenced  the  results  of  the  study  was  the  fact  that  the  functions  within  the 
VCVS  were  presented  only  in  English.  As  previously  acknowledged,  it  was  the  francophone 
Experimental  participants  that  were  observed  to  particularly  struggle  with  and  even  commented  on  their 
difficulties  understanding  the  valve  functions.  Even  though  the  Control  and  Qualified  testing  groups  also 
had  francophone  participants,  these  participants  were  not  observed  to  struggle  as  greatly  as  those  in  the 
Experimental  group.  Regardless  of  who  was  observed  to  struggle  or  not,  a  participant’s  ability  to 
comprehend  the  functions  (presented  in  English)  in  the  VCVS  was  not  controlled  for  and  thus,  may  have 
influenced  the  results  of  the  study. 
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8.  Lessons  Learned 

The  following  sub-sections  identify  and  describe  the  lessons  learned  during  the  conduct  of  the  VC  VS 
study. 

Participant  Safety  Briefings 

In  addition  to  participants  reading  and  signing  consent  forms,  participants  were  required  to  participate  in  a 
safety  briefing  prior  to  their  participation  aboard  HMCS  Comer  Brook.  In  these  safety  briefings 
participants  learned  how  to  apply  and  utilize  both  the  EBS  and  the  EEBD.  As  a  safety  measure, 
participants  received  this  safety  briefing  so  that  if  the  unfortunate  event  of  an  emergency  occurred  aboard 
which  required  immediate  evacuation,  participants  would  be  properly  prepared  to  effectively  handle  the 
situation.  Anytime  human  participants  participate  in  studies  where  their  participation  puts  them  at  risk 
(i.e.,  while  aboard  operational  equipment),  the  experimenters  must  provide  safety  measures  to  mitigate 
risks  associated  with  their  participation.  However,  as  previously  mentioned,  what  was  unanticipated  was 
that  these  safety  briefings  provided  participants  with  an  opportunity  to  gain  additional  spatial  awareness. 
To  mitigate  incidental  learning  opportunities,  it  is  suggested  that  an  area  aboard  the  submarine  close  to 
the  main  access  hatch,  be  designated  as  the  area  in  which  all  safety  briefings  will  take  place. 

Participant  Reactions  to  Submarine  Environment 

Even  though  not  a  single  participant  in  this  experiment  admitted  to  being  claustrophobic  while  aboard 
HMCS  Comer  Brook,  several  participants  commented  upon  the  spatially  constrained  close  quarters  and 
compartments  within  which  the  submariners  had  to  live  and  work.  One  participant,  however,  commented 
upon  how  much  he  disliked  being  aboard  HMCS  Comer  Brook  (showing  signs  of  relief  when  escorted 
back  onto  the  jetty),  in  addition  to  commenting  upon  the  spatially  constrained  close  compartments  during 
his  participation  in  both  the  VCVS  familiarization  and  skill  acquisition  session  and  submarine  acquisition 
session.  Each  time  a  participant  expressed  his  dislike  and/or  commented  on  the  spatial  constraints 
associated  with  the  submarine  environment,  the  experimenters  paid  closer  attention  for  signs  and/or 
behaviours  associated  with  claustrophobia.  To  try  to  gain  a  sense  of  how  participants  were  adapting  to 
this  new  environment,  the  experimenters  asked  participants  while  aboard  the  submarine  how  they  were 
feeling  and  if  a  break  was  required.  The  lesson  learned  from  this  experience  was  that  when  subjecting 
participants  to  new  and  unfamiliar  environments  that  have  several  spatially  constrained  areas,  a 
participant’s  susceptibility  to  claustrophobia  needs  to  be  not  only  assessed  and  anticipated,  but  also  the 
experimenter  should  have  a  realistic  evacuation  plan  in  the  event  that  a  participant  were  to  experience 
such  fears. 

Operational  Equipment 

Another  lesson  learned  from  supporting  the  VCVS  study  is  that  no  matter  how  much  time  is  spent 
preparing  for  and  planning  how  to  execute  a  study,  when  conducting  testing  which  requires  real  world 
operational  equipment,  the  testing  protocol  and  the  testing  schedule  may  be  affected  by  unanticipated 
events  and  obstacles  reducing  the  operational  equipment’s  accessibility. 

During  the  course  of  the  VCVS  study,  the  following  unanticipated  events  were  encountered  that  reduced 
access  to  HMCS  Comer  Brook: 
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•  Cold  moves  to  charge  submarine  batteries; 

•  Filming  events;  and, 

•  Slight  delays  due  to  rebalancing  the  submarine. 

While  these  all  resulted  in  slight  testing  delays  and  minimal  impacts  to  the  execution  of  the  study,  the 
important  lesson  learned  was  that  scheduling  ample  testing  time  between  participants  was  the  best 
defence  in  diminishing  the  effects  associated  with  delayed  access  to  HMCS  Comer  Brook. 
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9.  Comments  provided  By  non-participants 

In  addition  to  the  participants  of  this  study,  C.  Kersten  showed  the  VCVS  to  two  experienced  RCN 
submariners.  Summarized  in  Table  9-1  below,  are  the  general  comments  made  about  the  VCVS  during 
these  two  separate  exposure  sessions. 

Table  9-1:  Comments  provided  by  non-participant  during  VCVS  exposure  sessions. 


Non-participant  Comments 


Simulation  graphics  were  very  realistic. 

Details  (little  specifics)  upon  equipment  and  various  systems  were  amazing  and  well  done. 

The  relative  (proportional)  dimensioning  of  equipment  and  systems  seemed  to  be  quite  accurate  and 
realistic  to  real  life  environment. 

The  “snake  pit”  area  seemed  to  be  empty  and  missing  a  lot  of  extra  things  (i.e.  equipment,  systems  and 
valves)  compared  to  real  life  “snake  pit”  area. 

Indicated  that  the  VCVS  in  the  current  state  (i.e.,  having  areas  that  are  not  completely  developed  and 
thus  not  entirely  representative  of  the  real  boat  areas  such  as  snake  pit),  may  mislead  individuals  upon 
how  easy  it  is  to  locate  particular  valves  because  they  are  not  camouflaged  within  their  true  complex 
environments  (i.e.,  hidden  by  other  valves,  objects  and/or  equipment). 

Believed  that  on-board  training  still  remains  the  best  method  to  train  submariners. 

Believed  that  in  the  absence  of  the  gaining  access  to  the  real  boat  (submarine),  the  VCVS  could  be 
potentially  used  as  a  training  platform  and  quite  possibility  be  the  next  best  thing  to  actual  physical 
access  to  the  boat  (submarine). 

Indicated  that  he  could  see  the  potential  possibilities  of  expanding  the  VCVS  to  include  additional  tasks 
within  different  areas  of  the  boat  (submarine). 

Expressed  the  RCN’s  need  for  a  boat  (submarine)  physical/  simulation  based  training  platform  in  the 
near  future. 
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10.  Conclusion 

Based  upon  my  experiences  and  observations  during  the  VC  VS  familiarization  and  skill  acquisition 
sessions,  and  my  personal  experience  training  with  the  VCVS,  I  believe  the  VCVS  is  sufficiently 
sophisticated  to  provide  general  spatial  training  of  the  Victoria  Class  submarine.  However,  there  are 
opportunities  to  improve  the  VCVS  as  indicated  by  the  recommendations  put  forth  in  this  contract  report. 
Such  improvements  are  expected  to  improve  the  effectiveness  for  training  personnel  on  procedural  tasks 
associated  with  the  Victoria  Class  Submarine. 
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12.  acronyms  and  abbreviations 

The  following  list  identifies  the  acronyms  and  abbreviations  used  throughout  this  technical  report: 


A.T.U. 

Air  Treatment  Unit 

CAE  PS 

CAE  Professional  Services  (Canada)  Inc 

CFB 

Canadian  Forces  Base 

DMTE 

Director  Maritime  Training  and  Education 

DR  DC 

Defence  Research  and  Development  Canada 

DWS 

Duty  Watch  Supervisor  (DWS) 

EBS 

Emergency  Breathing  System 

EEBD 

Emergency  Escape  Breathing  Device 

FWD 

Forward 

HMC 

Her  Majesty’s  Canadian 

HMCS 

Her  Majesty’s  Canadian  Ship 

HREC 

Human  Research  Ethics  Committee 

HSI 

Human  Systems  Integration 

HTA 

Hierarchical  Task  Analysis 

NeLCoE 

Canadian  Navy  eLearning  Centre  of  Expertise 

PATs 

Personnel  Awaiting  Training 

RCN 

Royal  Canadian  Navy 

vcvs 

Victoria  Class  Virtual  Submarine 
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Tasks  Executed  by  CAE  Ps  ThrougHout  project 

The  following  list  identifies  the  tasks  fulfilled  by  CAE  PS  during  the  conduct  of  this  project: 

a)  Attended  project  kick  off  meeting  at  DRDC  Toronto; 

b)  Completed  and  obtained  certification  from  “Tri-Council  Policy  Statement:  Ethical  Conduct  for 
Research  Involving  Humans  Course  on  Research  Ethics  (TCPS  2:  CORE); 

c)  Learned  how  to  isolate  Bulkhead  35  from  both  the  forward  and  aft  compartments; 

d)  Developed  and  revised  marking  criteria  templates  for  evaluating  task  performance  in  the  VCVS  and 
aboard  HMCS  Comer  brook; 

e)  Conducted  informal  pilots  at  DRDC  Toronto; 

f)  Prepared  and  gathered  all  documents  required  for  study  (marking  sheet,  consent  forms,  payment 
forms,  instruction  sheets  and  participant  information  sheets); 

g)  Trained  by  Subject  Matter  Experts  (SMEs)  upon  isolating  bulkhead  35  (from  both  the  forward  and 
aft  compartments)  on  EIMCS  Comer  Brook; 

h)  Created  monthly  progress  reports; 

i)  Conducted  experiment  testing  sessions  aboard  EIMCS  Comer  Brook  and  in  Classroom  N60  on  the 
Canadian  Forces  Base  (CEB)  Esquimalt,  British  Columbia  (Qualified,  Experimental  and  Control 
groups); 

i.  Briefed  participants  upon  study  roles  and  rights  associated  with  their  participation; 

ii.  Asked  participants  to  complete  consent  and  payment  forms; 

iii.  Trained  participants  how  to  isolation  bulkhead  35  from  the  forward  and  aft  compartments  (in 
both  the  VCVS  and  aboard  EIMCS  Corner  Brook); 

iv.  Manually  testing  and  recording  participant  performance  data  during  testing  sessions; 

j)  Coded  and  entered  manually  recorded  performance  data  into  excel  files; 

k)  Coordinated  performance  data  backups  (VCVS  generated  log  files  and  performance  data  excel  files); 

l)  Returned  experimental  laptops  to  IT  Support  department  on  CEB  Esquimalt,  British  Columbia; 

m)  Attended  a  post  study  meeting  at  DRDC  Toronto  to; 

i. deliver  participant  performance  data  sheets  (data  manually  recorded  during  study)  and  a  CD 
which  contained  all  performance  data  excel  files  from  all  30  participants; 
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ii.  converse  over  preliminary  results  obtained;  and 

iii.  discuss  summary  report  expectations  (prepared  and  verified  “Table  of  Contents”); 

n)  Performed  descriptive  statistics  upon  participant  information; 

o)  Developed  and  delivered  customized  draft  technical  report; 

p)  Revised  draft  technical  report  based  upon  client  feedback  received; 

q)  Submitted  final  technical  report;  and 

r)  Attended/ participated  in  meeting  with  client  and  the  Canadian  Navy  eLearning 
Centre  of  Expertise  (NeLCoE). 
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Aft  compartment  VCVS  Score  Sheet 


SIMULATION  :  AFT  DECK  VALVES 

Session  # _  Starting  Location: 

Tri„.  u  Participant  ID  #: _ 


Tasks 

(No  Order) 

Participant  Action/  Verbalization 

Error 

Help 

Correct 

Notes 

Path/  Location 

1)  Shut  off  W608 

1.1)  Locate  and  pick  up  ratchet  for 
W608 

Click  on  ratchet 

1 .2)  Use  ratchet  to  shut  off  valve 
VV608 

Locate  W608 

Click  on  W608 

ID  W608 

Function:  FWD  Battery  Ventilation  Shut 

1.3)  Replace  ratchet 

Click  on  ratchet 

Path/  Location 

2)  Shut  off  W607 

2.1)  Shut  off  valve  W607 

Click  on  W607 

ID  W607 

Function:  Ventilation  Supply  Isolation 

Shut 

3)  Shut  off  W606 

3.1)  Shut  off  valve  W606 

Click  on  W606 

ID  W606 

Function:  Bulkhead  Exhaust  Isolation  Shut 

Path/  Location 

4)  Shut  off  W803 

4.1)  Shut  off  valve  W803 

Click  on  W803 

ID  W803 

Function:  BH35  Sonar  Cab  Isolation  Shut 
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SIMULATION  :  AFT  DECK  VALVES 

Session  # _  Starting  Location: 

Xri„i  u  Participant  ID  #: _ 


Tasks 

(No  Order) 

Participant  Action/  Verbalization 

Error 

Help 

Correct 

Notes 

Path/  Location 

5)  Shut  off  LPB803 

5.1)  Open  LPB803  cupboard 

Click  on  LPB803  cupboard  door 

5.2)  Enter  cupboard 

Press  "E"  key  to  enter  cupboard 

5.3)  Pick  up  ratchet 

Click  on  ratchet  for  LPB803 

5.4)  Use  ratchet  to  shut  off  valve 
LPB803 

Click  on  LPB803 

ID  LPB803 

Function:  LP  Blower  Suction  from  WSC 
Shut 

5.5)  Replace  ratchet 

Click  on  ratchet 

5.6)  Exit  the  LPB803  cupboard 

Press  "E"  key  to  exit  cupboard 

5.7)  Close  LPB803  cupboard 

Click  on  LPB803  cupboard  door 

Path/  Location 

6)  Shut  off  W801 

6.1)  Shut  off  valve  W801 

Click  on  W801 

ID  W801 

Function:  WSC  Supply  Isolating  Shut 
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forward  (FWD)  compartment  VC  VS  score  sheet 


SIMULATION  :  FWD  DECK  VALVES 

Session  # _  Starting  Location: _ 

tv;.,,  u  Participant  ID  #: _ 


Tasks 
(No  Order) 

Participant  Action/  Verbalization 

Error 

Help 

Correct 

Notes 

Path/  Location 

1)  Shut  off  W608 

1.1)  Locate  and  pick  up  ratchet 

VV608 

Click  on  ratchet 

1 .2)  Use  ratchet  to  shut  off  valve 
VV608 

Locate  VV608 

Click  on  W608 

ID  W608 

Function:  FWD  Battery  Ventilation  Shut 

1.3)  Replace  ratchet 

Click  on  ratchet 

Path/  Location 

2)  Shut  off  W607 

2.1)  Shut  off  valve  W607 

Click  on  W607 

ID  W607 

Function:  Ventilation  Supply  Isolation 

Shut 

3)  Shut  off  W606 

3.1)  Shut  off  valve  W606 

Click  on  W606 

ID  W606 

Function:  Bulkhead  Exhaust  Isolation  Shut 

Path/  Location 

4)  Shut  off  W803 

4.1)  Shut  off  valve  W803 

Click  on  W803 

ID  W803 
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SIMULATION  :  FWD  DECK  VALVES 

Session  # _  Starting  Location: _ 

Tt.i„i  u  Participant  ID  #: _ 


Tasks 
(No  Order) 

Participant  Action/  Verbalization 

Error 

Help 

Correct 

Notes 

Function:  BH35  Sonar  Cab  Isolation  Shut 

Path/  Location 

5)  Shut  off  LPB803 

5.1)  Shut  off  valve  LPB803 

Click  on  LPB803 

ID  LPB803 

Function:  LP  Blower  Suction  from  WSC 
Shut 

Path/  Location 

6)  Shut  off  W801 

6.1)  Locate  and  pick  up  ratchet 

VV801 

Click  on  ratchet 

6.2)  Use  ratchet  to  shut  off  valve 

VV801 

Click  on  W801 

ID  W801 

Function:  WSC  Supply  Isolating  Shut 

6.3)  Replace  ratchet 

Click  on  ratchet 
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Aft  compartment  Submarine  score  sheet 


SUBMARINE  :  AFT  DECK  VALVES 

Session  # _  Starting  Location: 

Tri„.  u  Participant  ID  #: _ 


Tasks 
(No  Order) 

Participant  Action/  Verbalization 

Error 

Help 

Correct 

Notes 

Path/  Location 

1)  Shut  off  W608 

1.1)  Locate  and  pick  up  ratchet 

ID  ratchet 

1.2)  Locate  VV608 

ID  W608 

1 .3)  Use  ratchet  to  shut  off  valve 
W608 

Connect  ratchet  on  valve  W608 

Use  ratchet  to  turn  valve  W608 

Function:  FWD  Battery  Ventilation  Shut 

1.4)  Replace  ratchet 

Disconnect  ratchet  from  valve  W608 

Return  ratchet  to  proper  location 

Path/  Location 

2)  Shut  off  W607 

2.1)  Shut  off  valve  W607 

ID  W607 

Turn  valve  W607 

Function:  Ventilation  Supply  Isolation 

Shut 

3)  Shut  off  W606 

3.1)  Shut  off  valve  W606 

ID  W606 

Turn  valve  VV606 

Function:  Bulkhead  Exhaust  Isolation  Shut 

Path/  Location 

4)  Shut  off  W803 

4.1)  Shut  off  valve  W803 

ID  W803 

Turn  valve  W803 
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CAE 


SUBMARINE  :  AFT  DECK  VALVES 

Session  # _  Starting  Location: 

Xri„i  u  Participant  ID  #: _ 


Tasks 
(No  Order) 

Participant  Action/  Verbalization 

Error 

Help 

Correct 

Notes 

Function:  BH35  Sonar  Cab  Isolation  Shut 

Path/  Location 

5)  Shut  off  LPB803 

5.1)  Open  LPB803  cupboard 

Open  LPB803  cupboard  door 

5.2)  Enter  cupboard 

Enter  LPB803  cupboard 

5.3)  Locate  and  pick  up  ratchet 

ID  ratchet 

5.4)  Use  ratchet  to  shut  off  valve 
LPB803 

ID  LPB803 

Connect  ratchet  on  LPB803 

Use  ratchet  to  turn  LPB803 

Function:  LP  Blower  Suction  from  WSC 
Shut 

5.5)  Replace  ratchet 

Disconnect  ratchet  from  LPB803 

Return  ratchet  to  proper  location 

5.7)  Exit  the  LPB803  cupboard 

Exit  cupboard 

5.8)  Close  LPB803  cupboard 

Close  LPB803  cupboard  door 

Path/  Location 

6)  Shut  off  W801 

6.1)  Shut  off  valve  W801 

ID  W801 

Turn  valve  VV801 

Function:  WSC  Supply  Isolating  Shut 
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Forward  (FWD)  compartment  submarine  score  sheet 


SUBMARINE  :  FWD  DECK  VALVES 

Session  # _  Starting  Location: _ 

Xri„i  u  Participant  ID  #: _ 


Tasks 
(No  Order) 

Participant  Action/  Verbalization 

Error 

Help 

Correct 

Path/  Location 

1)  Shut  off  W608 

1.1)  Locate  and  pick  up  ratchet 

ID  ratchet 

1.2)  Locate  VV608 

ID  W608 

1 .3)  Use  ratchet  to  shut  off  valve 
VV608 

Connect  ratchet  on  valve  W608 

Use  ratchet  to  turn  valve  W608 

Function:  FWD  Battery  Ventilation  Shut 

1 .4)  Replace  ratchet 

Disconnect  ratchet  from  valve  W608 

Return  ratchet  to  proper  location 

Path/  Location 

2)  Shut  off  W607 

2.1)  Shut  off  valve  W607 

ID  W607 

Turn  valve  W607 

Function:  Ventilation  Supply  Isolation 

Shut 

3)  Shut  off  W606 

3.1)  Shut  off  valve  W606 

ID  W606 

Turn  valve  W606 

Function:  Bulkhead  Exhaust  Isolation  Shut 

Path/  Location 

4)  Shut  off  W803 

4.1)  Shut  off  valve  W803 

ID  W803 

Turn  valve  VV803 

Function:  BH35  Sonar  Cab  Isolation  Shut 

Path/  Location 

5)  Shut  off  LPB803 

5.1)  Shut  off  valve  LPB803 

ID  LPB803 

Turn  valve  LPB803 

Function:  LP  Blower  Suction  from  WSC 
Shut 

Path/  Location 

6)  Shut  off  W801 

6.1)  Locate  and  pick  up  ratchet 

ID  ratchet 

6.2)  Use  ratchet  to  shut  off  valve 

W801 

ID  W801 

Connect  ratchet  on  W801 

Use  ratchet  to  turn  W801 

Function:  WSC  Supply  Isolating  Shut 

6.3)  Replace  ratchet 

Disconnect  ratchet  from  W801 
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SUBMARINE  :  FWD  DECK  VALVES 

Session  # _  Starting  Location: _ 

Xri„i  u  Participant  ID  #: _ 


Tasks 
(No  Order) 

Participant  Action/  Verbalization 

Error 

Help 

Correct 

Return  ratchet  to  proper  location 
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